Review

Semantic SLAM: A comprehensive survey of methods and applications

We begin by reviewing several key articles, majorly focused on semantic SLAM and scene understanding. To ensure comprehensive coverage of the relevant literature, we create a carefully structured search query, combining several terms associated with semantic SLAM and scene understanding. One such query used is: semantic SLAM, semantic SLAM with scene understanding, indoor scene understanding, and outdoor scene understanding. To enhance the search results, we additionally used author keywords like semantic SLAM for scene understanding-related papers.

To ensure a comprehensive and unbiased review, we adhered to a strict survey methodology with clearly defined inclusion and exclusion criteria as illustrated in Fig. 6. Articles were selected based on the following focused criteria:

To ensure a thorough and methodical exploration of the research landscape, a systematic literature review was conducted based on the above-mentioned criteria, and the final results of the selected papers is illustrated as bar plot in Fig. 7.

To enhance the breadth of our review, we also analyzed the related work sections of the initially selected articles. This allowed us to identify 123 additional papers relevant to SLAM techniques and scene understanding, bringing the total number of surveyed articles to 314. Additionally, the distribution of publication sources for the articles included in our survey is illustrated in Fig. 8. A significant majority of the reviewed works (84%) were published in peer-reviewed journals, reflecting the maturity and credibility of the research in the semantic SLAM domain. Conference papers accounted for 11%, while arXiv preprints contributed 3% and books comprised 2%. Although the proportion of arXiv articles is relatively small, those included in our study were carefully selected based on their high citation counts and demonstrated impact in the field, highlighting their relevance. After outlining the strict criteria we used to select articles, the next step is to provide the reader with the essential background of Semantic SLAM. This foundation will help contextualize the works we review later on.

Semantic SLAM systems can be categorized based on their primary sensing modalities, each offering distinct advantages for different applications and environments. Table 1 presents a comparison of various sensor-based approaches evaluated in this survey. The following subsections examine five major categories: monocular semantic SLAM, stereo semantic SLAM, RGB-D semantic SLAM, LiDAR semantic SLAM, multi-modal semantic SLAM, and incremental semantic SLAM.

This section focuses on the application of semantic SLAM in both dynamic indoor and outdoor environments, highlighting the unique challenges and solutions for each context. In dynamic indoor environments, semantic SLAM must account for rapidly changing objects and obstacles, while outdoor environments introduce challenges such as varying terrain, weather conditions, and large-scale dynamic elements. By capturing semantic information, robots can achieve more robust localization and mapping in these complex and unpredictable settings.

The TUM RGB-D dataset (Lin, Zhang et al., 2025, Sturm et al., 2012) is a popular benchmark for evaluating SLAM and visual odometry systems. It provides RGB-D images captured using a Microsoft Kinect sensor, along with ground truth poses obtained through a motion capture system. The dataset encompasses various indoor environments, including both static and dynamic scenes, offering a diverse and challenging test bed for visual SLAM algorithms. In this paper, the dataset sequences are used to evaluate the results discussed in Sections 8, 8.4, with the list of sequences detailed in Table 4.

The KITTI dataset (Geiger, Lenz, & Urtasun, 2012) is a widely recognized benchmark for evaluating computer vision and SLAM algorithms, particularly in the context of autonomous driving. It contains high-resolution stereo and LiDAR data acquired from a vehicle navigating through various environments, such as urban, rural, and highway settings. The dataset provides ground truth poses obtained from a GPS/IMU system, enabling precise evaluation of SLAM performance in real-world outdoor environments. The KITTI-360 dataset extends this benchmark with longer sequences, providing more comprehensive evaluation capabilities for semantic SLAM and scene understanding tasks in complex urban environments (Liao, Xie, & Geiger, 2023). The KITTI dataset is used to replicate the results discussed in Sections 8, 8.4, with the specific sequences outlined in Table 5.

The Bonn RGB-D Dynamic dataset (Palazzolo, Behley, Lottes, Giguère, & Stachniss, 2019a) is a key resource for advancing research in RGB-D SLAM. It consists of 24 dynamic sequences and 2 static sequences, capturing activities such as box manipulation and balloon play, designed to test SLAM algorithms in realistic, dynamic environments. Each sequence is accompanied by ground truth sensor poses obtained from an Optitrack Prime 13 motion capture system, along with a 3D point cloud of the static environment, recorded using a Leica BLK360 terrestrial laser scanner. The dataset is formatted similarly to the TUM RGB-D dataset, facilitating compatibility with existing evaluation tools.

These publicly available datasets were part of the Kimera-Multi project (Tian, Chang et al., 2023, Wan and Luo, 2025). They were gathered using Unitree A1 quadrupedal robots and Clearpath Robotics Jackal wheeled robots, both equipped with RealSense D455 RGB-D cameras, IMUs, and Velodyne 3D LiDAR. The sequences contain RGB images, depth images, compressed grayscale images, wheel odometry, and LiDAR point clouds. The data was recorded in various locations across MIT’s campus, including indoor and outdoor areas, underground tunnels, and an undergraduate dormitory. The diverse environments and revisited locations in these datasets make them particularly valuable for evaluating loop closure detection algorithms, where semantic features combined with traditional bag-of-words approaches have shown significant improvements in recognition accuracy (Sun, Wang, Ni, & Li, 2024).

This dataset was created using the Unity simulator, where humans are simulated as realistic 3D models with standard graphic assets. It is used for 2D segmentation tasks and was developed as part of the Kimera project. For benchmarking purposes, the simulator provides ground truth poses for both humans and objects. The dataset contains visual–inertia data for various scenes, both with and without dynamic objects, covering environments such as offices, apartments, subways, and neighborhoods (Rosinol et al., 2021).

These datasets consist of simulated urban outdoor scenes within the TESSE environment, designed similarly to the uHumans dataset. The simulated car is equipped with four monocular cameras positioned at the front, rear, left, and right. Additionally, the dataset includes ground truth pixel-wise semantic labels for precise analysis (Abate et al., 2024, Zhi et al., 2024). This multi-camera setup is particularly relevant for autonomous parking research, where similar sensor configurations have been successfully deployed in both outdoor valet parking scenarios (Abate et al., 2023) and indoor parking environments with semantic object detection capabilities (Shao, Zhang, Zhang, Shen, & Zhou, 2022).

This dataset is built for lifelong SLAM of service robots. The data is collected using two cameras: the RealSense D435i, which provides RGB-D images and IMU measurements, and the RealSense T265 tracking module, which captures stereo fisheye images and IMU measurements. The dataset includes five scenes, each containing 2 to 7 sequences taken at different times. Ground truth robot poses for each scene are provided by the Optitrack MCS and Hokuyo LiDAR systems (Shi, Li et al., 2020).

The BeVIS dataset specifically targets the challenging domain of indoor parking environments. This comprehensive benchmark provides ground-truth trajectories for evaluating SLAM systems in parking structures, where GPS-denied conditions and repetitive structural patterns pose unique challenges. The dataset supports the evaluation of tightly-coupled semantic SLAM frameworks that integrate front-view cameras, inertial sensors, and surround-view systems for robust localization and detection of semantic objects in parking scenarios (Shao et al., 2023).

The Scenes dataset v2 includes RGB and depth images from 14 scenes featuring various furniture items such as chairs, coffee tables, sofas, and tables, along with a selection of objects from the RGB-D Object dataset, including bowls, caps, cereal boxes, coffee mugs, and soda cans. Each scene has ground truth annotations and is represented as point cloud data, generated by aligning multiple video frames using Patch Volumes Mapping (Lai et al., 2014, Shao et al., 2025).

This dataset comprises RGB video sequences of 52 cars, captured with a camcorder in a full 360° rotation. Each video contains approximately 1500 to 3500 frames, which are uniformly downsampled to around 120 frames to accelerate the 3D reconstruction process (Sedaghat and Brox, 2015, Shao et al., 2025).

This dataset features a large and diverse collection of RGB-D and reconstructed models, ranging from shoes, mugs, and toys to grand pianos, construction vehicles, and large outdoor sculptures. The data was captured using PrimeSense Carmine cameras with a resolution of 640 × 480 pixels and a frame rate of $30\mathrm{Hz}$. Each scan consists of both color and depth images, with pixel values representing depth in millimeters (Choi, Zhou, Miller, & Koltun, 2016).

Further, to support both researchers and practitioners, we provide a summary of widely used open-source frameworks that extend traditional SLAM with semantic capabilities. Table 6 highlights prominent tools, their key features, and associated publications, offering practical resources for replicating results and advancing research in semantic SLAM.

The quality of the global map is important for accurate localization. To address this, Fan et al. proposed a novel semantic SLAM method that builds an accurate point cloud map while generating bounding boxes and masks using BlitzNet. The approach enables the creation of depth-stable points by accurately matching features in dynamic regions (Fan et al., 2020). Similarly, Han et al. provided a detailed review of indoor semantic mapping, covering aspects such as spatial mapping, semantic information acquisition, and map representation (Han, Li, Wang, & Zhou, 2021a). Chen et al. presented an extensive survey of semantic SLAM, detailing recent developments and analyzing the extraction and processing of semantic information using state-of-the-art datasets (Chen, Xiao et al., 2025).

Zhu et al. proposed a dense SLAM system called NICE-SLAM, which creates a hierarchical scene representation using local information. This representation, optimized with pre-trained geometric priors, enables detailed reconstruction of large indoor scenes while being more scalable, efficient, and robust (Zhu et al., 2022). Similarly, Wei et al. introduced DO-SLAM, a novel SLAM algorithm built upon ORB-SLAM2 and designed to enhance localization accuracy and system robustness in dynamic environments. By introducing outlier detection, this approach aims to mitigate the impact of dynamic objects on SLAM performance, improving both accuracy and reliability in challenging scenarios (Wei, Zhou, Duan, Liu and An, 2023). Additionally, Yu et al. presented DS-SLAM, which also extends ORB SLAM2 for highly dynamic environments. Their method uses five threads-tracking, semantic segmentation, local mapping, loop closing, and dense semantic map creation to improve the localization accuracy (Yu et al., 2018a). In a similar vein, Eslamian et al. proposed Det-SLAM, based on ORB SLAM3 and Detectron2, which identifies and eradicates dynamic spots to accomplish semantic SLAM in dynamic situations (Eslamian & Ahmadzadeh, 2022). Xu et al. further advanced this line of research with HMC-SLAM, a robust RGB-D SLAM system that leverages hierarchical multidimensional clustering to detect and filter dynamic features, significantly enhancing pose estimation in highly dynamic scenes (Xu, Zheng, Pan, & Yu, 2025). Moreover, Kim et al. developed SimVODIS, a unified framework that simultaneously performs visual odometry, object detection, and instance segmentation in a single self-supervised architecture that enables both geometric and semantic understanding for downstream SLAM or perception tasks in complex environments (Kim, Kim, & Kim, 2022). These semantic-based approaches have evolved to include adaptive fusion mechanisms that assign dynamic probabilities to detected objects, allowing the system to intelligently adjust its reliance on semantic versus geometric information based on scene complexity (Jiao, Wang, Li, Deng, & Xu, 2022). The computational efficiency of semantic integration can be further improved through selective frame processing. For example, the semantic segmentation can be applied only to keyframes rather than every frame, achieving real-time performance without sacrificing localization accuracy (Lee, Back, Hwang, & Chun, 2023b). Lightweight visual odometry systems have taken efficiency further by integrating adaptive geometric-semantic feature processing that dynamically balances computational load based on scene dynamics. This approach enables robust performance even on resource-constrained platforms (Wei, Huang, Liu and Zhou, 2023). Beyond frame-level optimizations, deep learning approaches have also improved loop closure detection efficiency through weighted triplet loss functions that learn discriminative features for place recognition, reducing the computational burden of exhaustive frame matching (Dong et al., 2022).

Recent advancements in semantic SLAM for indoor environments, as highlighted by key research papers, underscore the integration of deep learning techniques like BlitzNet and Detectron2 to enhance semantic understanding and dynamic object detection within SLAM systems. Despite these strides, future research should prioritize further enhancing the robustness of SLAM systems in highly dynamic environments, optimizing them for real-time performance, and advancing the integration of multi-sensor data for improved accuracy and efficiency.

Recent approaches to outdoor semantic SLAM have emphasized robust handling of dynamic elements and environmental variability. Dynamic SLAM exemplifies this trend by integrating SegNet-based semantic segmentation with ORB-SLAM2, using spatial motion information to achieve a 39.5% accuracy improvement in challenging outdoor conditions (Wen et al., 2023). Lin et al. developed the DPL SLAM technique, which combines ORB SLAM3 with a line detector segment network for efficient pose estimation. They also incorporated CUDA-enabled YOLOv5 for object detection to extract semantic information and remove abnormal features. The authors claim that this novel algorithm excels by not relying on a single source of information, effectively handling both known and unknown dynamic objects (Lin, Zhang, Tian, Yu, & Lan, 2024). Similarly, Zhang et al. proposed a semantic-based visual SLAM technique using ORB SLAM3 with TensorRT-optimized YOLOX to detect humans and non-humans in both indoor and outdoor environments (Zhang & Li, 2023). RSO-SLAM by Qin et al. integrates instance segmentation and optical flow to enhance robustness and localization accuracy in dynamic scenarios. Using a “KMC kmeans + connectivity” algorithm and ORB SLAM2, it detects motion regions and effectively handles non-rigid objects and slow-moving targets. However, the system struggles when large moving objects dominate the field of view or when significant changes occur (Qin et al., 2024).

Li et al. proposed a VSLAM method based on ORB SLAM2 integrated with Deeplab v3+ that incorporates semantic information to eliminate the negative effects of dynamic objects on precise localization (Li, Song et al., 2023). Ai et al. developed a new stereo SLAM system that combines ORB SLAM2 with the deep learning model ENet to enhance the performance of camera pose and trajectory estimation. The author claims that this system is robust and practical, particularly in highly dynamic and complex urban environments (Ai et al., 2023b). Similarly, Esparza et al. proposed a stereo SLAM approach for both indoor and outdoor dynamic environments, using ORB SLAM2 with a neural network-based semantic segmentation and geometrical constraints to effectively eliminate dynamic objects (Esparza & Flores, 2022).

The synthesis of findings from papers on outdoor scene understanding highlights a notable shift towards integrating advanced semantic segmentation and deep learning techniques with traditional SLAM frameworks like ORB SLAM2 and ORB SLAM3. By incorporating methods such as Deeplab v3+, YOLOX, and ENet, these studies demonstrate significant enhancements in accurately localizing and mapping dynamic outdoor environments. This integration enables SLAM systems to effectively discern between stationary and moving objects, thus improving robustness in complex urban landscapes and varied outdoor conditions. However, challenges persist in scenarios involving large moving objects and rapid environmental changes, highlighting the ongoing need for further research. Overall, the fusion of semantic information with SLAM technologies promises advancements in autonomous navigation and spatial understanding, crucial for applications ranging from robotics to augmented reality in outdoor environments. A timeline diagram illustrating the most commonly implemented semantic SLAM systems for both indoor and outdoor scenes is shown in Fig. 18.

Recent research in semantic SLAM has introduced new techniques that significantly change how systems understand and build maps of the environment. One major development is the use of a method called 3D Gaussian Splatting (3DGS), which allows systems to create detailed and semantically rich 3D maps much faster and more accurately than before. For example, methods like SGS-SLAM (Li, Liu et al., 2024, Yang, Wang et al., 2025) and OpenGS-SLAM (Chen, Zhang et al., 2025, Guerrero-Font et al., 2021, Li, Gu et al., 2018, Yang, Gao et al., 2025) use 3DGS to represent scenes using small 3D blobs (called Gaussians) instead of traditional pixels or point clouds. This leads to sharper object boundaries, faster map updates, and more precise object detection in the environment. Unlike older methods that use slow and blurry representations (like NeRF), these new techniques are more efficient and suitable for real-time use.

In addition, Hier-SLAM++ (Li, Hao et al., 2025, Zhang, Guo et al., 2024) goes a step further by combining 3DGS with powerful AI models such as SAM and CLIP, which have been trained on large datasets. This combination helps the system recognize and label a wide range of objects, even in unfamiliar environments, and works with both RGB-D and simple monocular camera inputs. These innovations make semantic SLAM systems smarter, faster, and more adaptable to real-world challenges.

Another emerging direction is the fusion of vision–language models with SLAM systems. For example, FindAnything (Abdelnasser et al., 2016, Laina et al., 2025) introduces an open-vocabulary SLAM framework that supports natural language queries during mapping, marking a shift toward generalizable, interactive scene understanding.

Additionally, SG-SLAM (Chen, Li et al., 2025, Dube et al., 2020, Wang, Lu et al., 2025) introduces a semantic-graph-enhanced LiDAR SLAM system. Instead of relying on point-wise labels, SG-SLAM constructs a robust object-level semantic graph, enhancing re-localization, loop closure, and global map consistency. It achieves real-time performance across challenging LiDAR datasets such as KITTI, MulRAN, and Apollo, outperforming both geometry-based and semantic SLAM baselines.

Collectively, these works demonstrate a move toward explicit, modular, and generalizable representations in semantic SLAM. They enable not only improved accuracy and robustness but also new capabilities such as zero-shot segmentation, open-set reasoning, and interactive manipulation, which are critical for next-generation applications in robotics, embodied AI, and autonomous systems. These developments illustrate how the field is progressing, but to truly assess their impact we must also consider the evaluation criteria used in the semantic SLAM. Therefore, the following section focuses on the performance metrics applied in Semantic SLAM research.